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Abstract 



Much of the current theory of networked control systems uses simple point-to-point communication models as 
an abstraction of the underlying network. As a result, the controller has very limited information on the network 
conditions and performs suboptimally. This work models the underlying wireless multihop mesh network as a 
graph of links with transmission success probabilities, and uses a recursive Bayesian estimator to provide packet 
delivery predictions to the controller. The predictions are a joint probability distribution on future packet 
delivery sequences, and thus capture correlations between successive packet deliveries. We look at finite horizon 
LQG control over a lossy actuation channel and a perfect sensing channel, both without delay, to study how the 
controller can compensate for predicted network outages. 



1 Introduction 

Increasingly, control systems are operated over large-scale, networked infrastructures. In fact, several companies 
today are introducing devices that communicate over low-power wireless mesh networks for industrial automation 
and process control [TJ |5] . While wireless mesh networks can connect control processes that are physically spread 
out over a large space to save wiring costs, these networks are difficult to design, provision, and manage [3J 0]. 
Furthermore, wireless communication is inherently unreliable, introducing packet losses and delays, which are 
detrimental to control system performance and stability. 

Research in the area of Networked Control Systems (NCSs) [5] addresses how to design control systems which 
can account for the lossy, delayed communication channels introduced by a network. Traditional tasks in control 
systems design, like stability/performance analysis and controller/estimator synthesis, are revisited, with network 
models providing statistics about packet losses and delays. In the process, the studies highlight the benefits and 
drawbacks of different system architectures. For example, Figure [I] depicts the general system architecture of a 
networked control system over a mesh network proposed by Robinson and Kumar [6]. A fundamental architecture 
problem is how to choose the best location to place the controllers, if they can be placed at any of the sensors, 
actuators, or communication relay nodes in the network. One insight from Schenato et al. [7] is that if the controller 
can know whether the control packet reaches the actuator, e.g., we place the controller at the actuator, then the 
optimal LQG controller and estimator can be designed separately (the separation principle). 




Figure 1: A networked control system over a mesh network, where the controllers can be located on any node. 



To gain more insights on how to architect and design NCSs, two limitations in the approach of many current 
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NCS research studies need to be addressed. The first limitation is the use of simple models of packet delivery 
over a point-to-point link or a star network topology to represent the network, which are often multihop and more 
complex. The second limitation is the treatment of the network as something designed and fixed a priori before the 
design of the control system. Very little information is passed through the interface between the network and the 
control system, limiting the interaction between the two "layers" to tune the controller to the network conditions, 
and vice versa. 

1.1 Related Works 

Schenato et al. [7] and Ishii [B] study stability and controller synthesis for different control system architectures, 
but they both model networks as i.i.d. Bernoulli processes that drop packets on a single link. The information 
passed through the interface between the network and the control system is the packet drop probability of the link, 
which is assumed to be known and fixed. Seiler and Sengupta [§] study stability and "Hoo controller synthesis when 
the network is modeled as a packet-dropping link described by a two-state Markov chain (Gilbert-Elliott model), 
where the information passed through the network-controller interface are the transition probabilities of the Markov 
chain. Elia [TU] studies stability and the synthesis of a stabilizing controller when the network is represented by an 
LTI system with stochastic disturbances modeled as parallel, independent, multiplicative fading channels. 

Some related work in NCSs do use models of multihop networks. For instance, work on consensus of multi-agent 
systems typically study how the connectivity graph(s) provided by the network affects the convergence of the 
system, and is not focused on modeling the links. Robinson and Kumar [6] study the optimal placement of a 
controller in a multihop network with i.i.d. Bernoulli packet-dropping links, where the packet drop probability is 
known to the controller. Gupta et al. |12j study how to optimally process and forward sensor measurements at 
each node in a multihop network for optimal LQG control, and analyze stability when packet drops on the links 
are modeled as spatially-independent Bernoulli, spatially-independent Gilbert-Elliott, or memoryless spatially- 
correlated processes^ Varagnolo et al. |13j compare the performance of a time-varying Kalman filter on a wireless 
TDMA mesh network under unicast routing and constrained flooding. The network model describes the routing 
topology and schedule of an implemented communication protocol, TSMP [TJ], but it assumes that transmission 
successes on the links are spatially-independent and memoryless. Both Gupta et al. [T2] and Varagnolo et al. [13] 
are concerned with estimation when packet drops occur on the sensing channel, and the estimators do not need to 
know network parameters like the packet loss probability. 

1.2 Contributions 

Our approach is a step toward using more sophisticated, multihop network models and passing more information 
through the interface between the controller and the network. Similar to Gupta et al. |12j . we model the network 
routing topology as a graph of independent links, where transmission success on each link is described by a two-state 
Markov chain. The network model consists of the routing topology and a global TDMA transmission schedule. 
Such a minimalist network model captures the essence of how a network with bursty links can have correlated 
packet deliveries |15j , which are particularly bad for control when they result in bursts of packet losses. Using this 
model, we propose a network estimator to estimate, without loss of information, the state of the network given 
the past packet deliveries^] The network estimate is translated to a joint probability distribution predicting the 
success of future packet deliveries, which is passed through the network-controller interface so the controller can 
compensate for unavoidable network outages. The network estimator can also be used to notify a network manager 
when the network is broken and needs to be reconfigured or reprovisioned, a direction for future research. 

Section [2] describes our plant and network models. We propose two network estimators, the Static Independent 
links, Hop-by-hop routing, Scheduled (SIHS) network estimator and the Gilbert-Elliott Independent links, Hop- 
by-hop routing, Scheduled (GEIHS) network estimator in Section [3] Next, we design a finite-horizon, Future- 
Packet-Delivery-optimized (FPD) LQG controller to utilize the packet delivery predictions provided by the network 

ifiere, "spatially" means "with respect to other links." 

2 Strictly speaking, we obtain the probability distribution on the states of the network, not a single point estimate. 
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Figure 2: A control loop for plant 7 with the network on the actuation channel. The network estimator N passes 
packet delivery predictions / h+n-x to the FPD controller C, with past packet delivery information obtained from 

k 

the network Jsf over an acknowledgement (ACK) channel. 



estimators, presented in Section [3] Section [5] provides an example and simulations demonstrating how the GEIHS 
network estimator combined with the FPD controller can provide better performance than a classical LQG controller 
or a controller assuming i.i.d. packet deliveries. Finally, Section [7] describes the limitations of our approach and 
future work. 



2 Problem Formulation 

This paper studies an instance of the general system architecture depicted in Figure [T] with a single control loop 
containing one sensor and one actuator. One network estimator and one controller are placed at the sensor, and we 
assume that an end-to-end acknowledgement (ACK) that the controller-to-actuator packet is delivered is always 
received at the network estimator, as shown in Figure [2] For simplicity, we assume that the plant dynamics are 
significantly slower than the end-to-end packet delivery deadline, so that we can ignore the delay introduced by the 
network. The general problem is to jointly design a network estimator and controller that can optimally control the 
plant using our proposed SIHS and GEIHS network models. In our problem setup, the controller is only concerned 
with the past, present, and future packet delivery sequence and not with the detailed behavior of the network, 
nor can it affect the behavior of the network. Therefore, the network estimation problem decouples from the 
control problem. The information passed through the network-controller interface is the packet delivery sequence, 
specifically the joint probability distribution describing the future packet delivery predictions. 

2.1 Plant and Network Models 

The state dynamics of the plant 7 in Figure [2] is given by 

x k+1 = Ax k + v k Bu k + w k , (1) 

where A € R exe , B € E , and w k are i.i.d. zero-mean Gaussian random variables with covariance matrix 
R w € §+, where S^j_ is the set of £ x I positive semidefinite matrices. The initial state xo is a zero-mean Gaussian 
random variable with covariance matrix R € Si and is mutually independent of w k . The binary random variable 
v k indicates whether a packet from the controller reaches the actuator (y k — 1) or not (y k = 0), and each v k is 
independent of xq and w k (but the v k s are not independent of each other). 
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Figure 3: The packet containing the control input Uk is generated right before time slot tk- The packet may be in 
transit through the network in the shaded time slots, until right before time slot t' k . Thus, time tk is aligned with 
the beginning of the time slot. 



Let the discrete sampling times for the control system be indexed by k, but let the discrete time for schedule 
time slots (described below) be indexed by t. The time slot intervals are smaller than the sampling intervals. The 
time slot when the control packet at sample time k is generated is denoted tk, and the deadline for receiving the 
control packet at the receiver is t' k . We assume that t' k < tk+i for all k. Figure [3] illustrates the relationship between 
t and k. 

The model of the TDMA wireless mesh network (N in Figure [2| consists of a routing topology G, a link model 
describing how the transmission success of a link evolves over time, and a fixed repeating schedule F^ T ). The SIHS 
network model and the GEIHS network model only differ in the link model. Each of these components will be 
described in detail below. 

The routing topology is described by G = (V,£), a connected directed acyclic graph with the set of vertices 
(nodes) V = {1, . . . ,M} and the set of directed edges (links) £ C : i,j € V, i ^ j}, where the number of 

edges is denoted E. The source node is denoted a and the sink (destination) node is denoted b. Only the destination 
node has no outgoing edges. 

At any moment in time, the links in G can be either be up (succeeds if attempt to transmit packet) or down 
(fails if attempt to transmit packet). Thus, there are 2 E possible topology realizations G — (V, £), where £ C £ 
represents the edges that are upj^] 

At time tk , the actual state of the topology is one of the topology realizations but it is not known to the network 
estimator. With some abuse of terminology, we define G< fc > to be the random variable representing the state of the 
topology at time ty^\ 

This paper considers the network under two link models, the static link model and the Gilbert-Elliott (G-E) 
link model. Both network models assume all the links in the network are independent. 

The static link model assumes the links do not switch between being up and down while packets are sent through 
the network. Therefore, the sequence of topology realizations over time is constant. While not realistic, it leads to 
the simple network estimator in Section [3. 1| for pedagogical purposes. The a priori transmission success probability 
of link I = is pi. 

The G-E link model represents each link I by the two-state Markov chain shown in Figure [4] At each sample 
time k, a link in state (down)transitions to state 1 (up) with probability pf, and a link from state 1 transitions 
to state with probability pfrj The steady-state probability of being in state 1, which we use as the a priori 
probability of the link being up, is 

Pl=pfM+pf) ■ 

The fixed, repeating schedule of length T is represented by a sequence of matrices F^ T ^ = (F^\ F^ 2 \ . . . , F^), 
where the matrix F^ -1 ( mod T ) +1 ) represents the links scheduled at time t. The matrix F^ e {0, l} MxM is defined 
from the set J 7 ^ C £ containing the links scheduled for transmission at time t. We assume that nodes can only 
unicast packets, meaning that for all nodes i, if € J 7 ^ then for all v ^ j,(i,v) J 7 ^. Furthermore, a 

3 Symbols with a tilde (~) denote values that can be taken on by random variables, and can be the arguments to probability 
distribution functions (pdfs). 

4 Strictly speaking, C' is a function mapping events to the set of all topology realizations, not to the set of real numbers. 
5 We can easily instead use a G-E link model that advances at each time step t, but it would make the following exposition and 
notation more complicated. 
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Figure 4: Gilbert-Elliott link model 



node holds onto a packet if the transmission fails and can retransmit the packet the next time an outgoing link is 
scheduled (hop-by- hop routing). Thus, the matrix has entries 

{1 if G or 

if i = j and Vw G V, (i, v) <£ J**) 
otherwise. 

An exact description of the network consists of the sequence of topology realizations over time and the schedule 
F*- T ). Assuming a topology realization G, the links that are scheduled and up at any given time t are represented 
by the matrix G {0, l} MxM , with entries 

fi if (»,j)e jMni, or 

fJ ;G) = J if i = j and Vu G V, (i, «) £ n £ (2) 
[ otherwise. 

Define the matrix F^*'^ = ^(*;<5)^(*+i;G) . ..p{t';G)^ guch that entry p^ 1 '^) i s 1 if a. packet at node i at 
time t will be at node j at time i', and is otherwise. Since the destination b has no outgoing links, a packet sent 

from the source a at time t reaches the destination b at or before time t' if and only if F^'* — 1. To simplify 
the notation, let the function S K indicate whether the packet delivery v G {0, 1} is consistent with the topology 
realization G, assuming the packet was generated at t K , i.e., 

fi if u - p(*- t -< 5 ) 

5 K (9;G) = \ 1 ■ (3) 

10 otherwise. 

The function assumes the fixed repeating schedule F^ T \ the packet generation time t K , the deadline t' K , the source 
a, and the destination b are implicitly known. 



2.2 Network Estimators 

As shown in Figure [2] at each sample time k the network estimator N takes as input the previous packet delivery 
Vk-i, estimates the topology realization using the network model and all past packet deliveries, and outputs the joint 
probability distribution of future packet deliveries f v h+s-i . For clarity in the following exposition, let V K G {0, 1} 

be the value taken on by the packet delivery random variable v K at some past sample time k. Let the vector 
VJT 1 = [V , . . . , Vk-i] denote the history of packet deliveries at sample time k, the values taken on by the vector 
of random variables fo _:1 = [^0, • • • , ^fe-i]- Then, 

Lt+B-xp?- 1 ) = PK +f, - x = uo"- 1 ^- 1 = vr 1 ) (4) 

k 

is the prediction of the next H packet deliveries, where ul +H ^ x = . . . , v^+h-i] is a vector of random variables 
representing future packet deliveries and C 1 " 1 G {0, 1} H ■ 



Figure 5: Graphical model describing the network estimation problem, is the measurement output variable at 
time k, and G< fc > is the hidden state of the network. 



The SIHS and GEIHS network estimators only differ in the network models. The parameters of the network 
models — topology G, schedule F^ T \ link probabilities {pi}ie£ or {pf,pf }i<=£, source a, sink b, packet generation 
times tk, and deadlines t' k — are known a priori to the network estimators and are left out of the conditional 
probability expressions. 

In Section |3j we will use the probability distribution on the topology realizations (our network state estimate), 

P(G< fc > = G\vt 1 = V^ 1 ) , 
to obtain / h+b-i from V _1 and the network model. 

2.3 FPD Controller 

The FPD controller (C in Figure [2| optimizes the control signals to the statistics of the future packet delivery 
sequence, derived from the past packet delivery sequence. We choose the optimal control framework because the 
cost function allows us to easily compare the FPD controller with other controllers. The control policy operates 
on the information set 

Ik ={xSX- 1 X" 1 } ■ (5) 
The control policy minimizes the linear quadratic cost function 

E {x'l 1 Qax N +Y:™Za xlQ 1 x n + v n ulQ 2 u n ] , (6) 

where Qo, Q%, and Q2 are positive definite weighting matrices and N is the finite horizon, to get the minimum cost 

J= min E[xTq x n +J2"=o x^Q 1 x n +v, 1 ,ulQ2u n ] 

«p,...,ujV_l 

Section [4] will show that the resulting architecture separates into a network estimator and a controller which uses 
the pdf f^k+H-i supplied by the network estimator (3\f in FigureJ^ to find the control signals u/-. 

3 Network Estimation and Packet Delivery Prediction 

We will use recursive Bayesian estimation to estimate the state of the network, and use the network state estimate 
to predict future packet deliveries. Figure [5] is the graphical model / hidden Markov model |16j describing our 
recursive estimation problem. 
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3.1 SIHS Network Estimator 

The steps in the SIHS network estimator are derived from Q. We introduce new notation for conditional pdfs (i.e., 
afc, (3k, Zk), which will be used later to state the steps in the estimator compactlyPl First, express f u k+H-i (j>" _1 ) 
as 



G 

i k+H-i (j>a _1 ) «fc(i'(f _1 ;G) Ph(G) 

where we use the relation 

POr^ 1 =j/ H - 1 |G( f! - 1 ) = G,vt 1 = Vt x ) =^{vt +B - 1 =u u - 1 \Gv°-v = G) . 

This relation states that given the state of the network, future packet deliveries are independent of past packet 
deliveries. The expression P(i^ +H_1 = u"~ 1 \G( k - 1 '> — G) indicates whether the future packet delivery sequence 
i>" _1 is consistent with the graph realization G, meaning 

H-l 



= u^lG^-v =G)=U 6 k+h (9 h ;G) , 



/i=0 



where Y[ is the and operator (sometimes denoted /\). The network state estimate at sample time k from past 
packet deliveries is fi k (G) and is obtained from the network state estimate at sample time k — 1, since 



<5 fc _i(V fc _ i; G) A=-i(G) 
*> . . -~ 



pr™-) - ri„*-> - v-m - p(l/fc ~ 1 = Vfe - 1 i G(fc ' 1) = G)-n&-^ = GK~ 2 = vr 2 ) 



Here, P(G<*-« = GK~ 2 = VJT 2 ) = P(G< fc - 2 > = GK~ 2 = V fc - 2 ) = /3 fe _i(G) for the static link model because 
G^- 1 ' = G< fc - 2 > = G<°). Again, we used the independence of future packet deliveries from past packet deliveries 
given the network state, 

PK-i = V k -i\G^ =G,^- 2 = V r r 2 ) =P(i/ fc _i = V*_i|G<»-u =G) . 

Note that P(^fc_i = Vfc-ilGc 1 - 1 ) = G) can only be or 1, indicating whether the packet delivery is consistent with 
the graph realization. Finally, F(i/ k -i = Vk-iWo ' 2 — ^cT 2 ) is the same for all G, so it is treated as a normalization 
constant. 

At sample time k = 0, when no packets have been sent through the network, /3q(G) = P(G<°> = G), which is 
expressed in (8d) below. This equation comes from the assumption that all links in the network are independent. 

To summarize, the SIHS Network Estimator and Packet Delivery Predictor is a recursive Bayesian estimator 
where the measurement output step consists of 

f^+*-ipr i )=^a k pr i \&)'0k{G) (8a) 

G 
H-l 

a k (v?- 1 ;G)= H 6 k+h (D h ;G) , (8b) 

h=0 

6 A semicolon is used in the conditional pdfs to separate the values being conditioned on from the remaining arguments. 
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and the innovation step consists of 

MC) - fc -' v »f (80 

mg)= m w I n 1 -pt) . ( 8d ) 

where and are functions, is a normalization constant such that (3 k (G) = 1, and the functions <5fc_|_^ 
and <5i._i arc defined by ^. 

3.2 GEIHS Network Estimator 

For compact notation in the probability expressions below, we use Vo" 1 in place of i/* -1 = VJ -1 and only write 
the random variable and not its value ( T ) . 

The derivation of the GEIHS network estimator is similar to the previous derivation, except that the state of 
the network evolves with every sample time k. Since all the links in the network are independent, the probability 
that a given topology G at sample time k — 1 transitions to a topology G after one sample time is given by 

r(G; g') = p(g« \G ( - h ~^) = | n i ~Ph) ( n pi) ( n pt) ( n i -pi) • w 

\he£'n£ ) \i 2 e£'\£ j \he£\£' j \i 4 ££\(£'u£) / 

First, express f^+H-i^"^ 1 ) as 

pk+^ivs- 1 ) = nvk+H-i \ G^ H -v) ; nvt +H -\G^"-v\v*- 1 ^ , 

S * ' G<. k + H -^ ' S * ' 

k 

where for h = 2, . . . , H — 1 

V(vt +h - 1 ,GV'+v\V5- 1 )= V P(G< fc + fe )|G< fe +'- 1 ))-P(^ + , l _i|G( fc + ft - 1 ))-P(^ + ^ 2 ,G( fc + fc - 1 )|VS- 1 ) . (10) 

" * ' G C*+*-V ~^ -~ ' " * ' 

ahik-AO^- 1 ,G h ) r(G fe ;G h _!) 6 k+h _ 1 t ;G h _ 1 ) a, l _ 1 , fe _ 1 (P h - 2 ,G h -i) 



When ft = 1, replace P(t/£ +h - 3 , G( fc +'-D IVfT 1 ) in (jTof with P(G( fc )|V^ 1 ) = P k \ k - X {G). The value P k \ k -i{G) comes 
from 

P(G«|Vo _1 ) = 51 P(G (fc) |G< fc - 1 ))-P(G fc - 1 '|VS- 1 ) . 



0k\k-i(G) r(G;G') k _ 1]k -i(G>) 



The value /3 k -i\ k -i(G') comes from (|7|), with /3 k replaced by P k -i\ k -i and f3 k -i replaced by /3 k -i\ k -2- Finally, 
A)l-i(G) = P(G<°)), where all links are independent and have link probabilities equal to their steady-state proba- 



bility of being in state 1, and is expressed in ( 1 If ) below. 

To summarize, the GEIHS Network Estimator and Packet Delivery Predictor is a recursive Bayesian estimator. 
The measurement output step consists of 

ful +H - l( y i> "' 1 ) = X! S k+H-i(vH-i;G H -x) ■ oiH-iik-ii^o _2 ,G ff _i) , (11a) 

where the function ajj-i\k-i 1S obtained from the following recursive equation for h = 2, . . . ,H — 1: 

ah\k-i(i'S~\Gh) = ^ T(G h ;G h -i) ■ S k+h -i(Ph-i;G h -i) ■ a h _ llk _ 1 (Do' 2 ,G h -i) , (lib) 
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with initial condition 



ai| fe -i(£ °,Gi) = X) r (Gi;G) -5 k (yo]G) ■ &| fe -i(G) 



The prediction and innovation steps consist of 



Pk-i\k-i(G) — 



Po\-i(G) 



flfc-iCVfc-i; G) ■ p k ^i\ k - 2 {G) 
Z k -i 




(11c) 

(lid) 
(lie) 
(llf) 



where affrifc-i, P k —i\ k —ii an d /^fcifc-i are functions, Z k _\ is a normalization constant such that P k —i\ k -i(G) = 1, 
and the functions S K (for the different values of k above) and T are defined by ([3| and ([9]), respectively. 

3.3 Packet Predictor Complexity 

The network estimators are trying to estimate network parameters using measurements collected at the border of 
the network, a general problem studied in the field of network tomography |17j under various problem setups. One 
of the greatest challenges in network tomography is getting good estimates with low computational complexity 
estimators. 

Our proposed network estimators are "optimal" with respect to our models in the sense that there is no loss of 
information, but they are computationally expensive. 

Property 1. The SIHS network estimator described by the set of equations ^ takes 0(E2 E ) to initialize and 
0(2 H 2 E ) to update the network state estimate and predictions at each step. The GEIHS network estimator described 
by the set of equations (11) takes 0(E2 2E ) to initialize and 0(2 H 2 2E ) for each update step. 



Proof. Let D = max^ t' k — t k . We assume that converting G to the set of links that are up, £, takes constant time. 
Also, one can simulate the path of a packet by looking up the scheduled and successful link transmissions instead of 

multiplying matrices to evaluate F^' trt,<3 \ so computing 5 K for each graph G only takes 0(D). The computational 
complexities below assume that the pdfs can be represented by matrices, and multiplying an £ x m matrix with a 
m x n matrix takes O(imn). 
SIHS packet delivery predictor complexity: 

Computing a k in (|8b[) takes 0(D2 H 2 E ) and computing fi k in 



takes 0(D2 E ), since there are 2 E graphs and 
2 H packet delivery prediction sequences. Computing / h+n-i in (8a) takes 0(D2 H 2 E ). The SIHS packet delivery 

k 

predictor update step is the aggregate of all these computations and takes 0(D2 H 2 E ). 

The initialization step of the SIHS packet delivery predictor is just computing /3q in pd} 
GEIHS packet delivery predictor complexity: 

Computing a ft | fc _i in ( |llb[ ) takes 0(D2 E + 2 E 2 h + 2 2E 2 h ) and computing ai| fe _i in ( fTTct takes 0{D2 E + 
2 e+i + 2^E+i)^ g0 compu ting all of them takes 0(DH2 E + 2 H (2 E + 2 2S )), or just 0{DH2 E + 2^2 2E ). Computing 
h\k-i in (fTTdt) takes 0{2 2E ) 



which takes 0(E2 l 



and computing (3 



fc-i|fc- 



i in (lie) takes 0(D2 ). Computing / h+h-i in (11a) takes 



0(D2 E + 2 H 2 E ). The GEIHS packet delivery predictor update step is the aggregate of all these computations and 
takes 0{DH2 E + 2 H 2 2E ). 

Computing V in Q takes 0(E2 2E ), and computing /3o|-i in (llf) takes 0(E2 E ). The initialization step of the 
GEIHS packet delivery predictor is the aggregate of these computations and takes 0(E2 2E ). 

If we assume that the deadline D is short enough to be considered constant, we get the computational com- 
plexities given in Property [l] □ 
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A good direction for future research is to find lower complexity, "suboptimal" network estimators for our problem 
setup, and compare them to our "optimal" network estimators. 

3.4 Discussion 

Our network estimators can easily be extended to incorporate additional observations besides past packet deliveries, 
such as the packet delay and packet path traces. The latter can be obtained by recording the state of the links 



that the packet has tried to traverse in the packet payload. The function S^-i in (8c) and (lie) just needs to be 
replaced with another function that returns 1 if the the received observation is consistent with a network topology 
G, and otherwise. The advantage of using more observations than the one bit of information provided by a packet 
delivery is that it will help the GEIHS network estimator more quickly detect changes in the network state. A 
more non-trivial extension of the GEIHS network estimator would use additional observations provided by packets 
from other flows (not from our controller) to help estimate the network state, which could significantly decrease 
the time for the network estimator to detect a change in the state of the network. This is non-trivial because the 
network model would now have to account for queuing at nodes in the network, which is inevitable with multiple 
flows. 



Note that the network state probability distribution, (3 k {G) in (8c) or f3k-i\k-i(G) in (lie), does not need 
to converge to a probability distribution describing one topology realization to yield precise packet predictions 
/^fc+ff-i (O"^ 1 ), where precise means there is one (or very few similar) high probability packet delivery sequence(s) 

v" 1 - Several topology realizations G may result in the same packet delivery sequence. 

Also, note that the GEIHS network estimator performs better when the links in the network are more bursty. 
Long bursts of packet losses from bursty links result in poor control system performance, which is when the network 
estimator would help the most. 



4 FPD Controller 

In this section, we derive the FPD controller using dynamic programming. Next, we present two controllers for 
comparison with the FPD controller. These comparative controllers assume particular statistical models (e.g., i.i.d. 
Bernoulli) for the packet delivery sequence pdf which may not describe the actual pdf, while the FPD controller 
allows for all packet delivery sequence pdfs. We derive the LQG cost of using these controllers. Finally, we present 
the computational complexity of the optimal controller. 

4.1 Derivation of the FPD Controller 

We first present the FPD controller and then present its derivation. 

Theorem 4.1. For a plant with state dynamics given by |l]) ; the optimal control policy operating on the information 
set ([5]) which minimizes the cost function ^ results in an optimal control signal u k = —L k x k , where 

L k = (O2 + B T S k+1 (u k ^ > - 1 )By 1 B T S k+1 (u k =i,ut 1 )A (12) 

and Sk ■ {0, l} fe 1— > S £ + and s k ■ {0, l} fe h- > R + are the solutions to the cost-to-go at time k, given by 

Shtf- 1 ) =Qi + A T E[S k+1 (^)\ut 1 ]A 



(v k = iK" 



A T S k+1 (u k =i,^)B(Q 2 + B T S k+ i{v k =i,v*- 1 )B) 1 B T S k+1 (u k =i,u^)A 



SfcK fc_1 )= trace {E[5 fe+ iK)|i/ fe 'ji?™} + E [s fc+ i("o )K _1 ] 
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Proof. The classical problem in Astrom [TS] is solved by reformulating the original problem as a recursive min- 
imization of the Bellman equation derived for every time instant, beginning with N. At time n, we have the 
minimization problem 



N-l 



E 



E 



min E[x n Q q xn + ^ xjQ\Xi + ViujQ 2 Ui] 

N-l 

min E[x N Q x N + } j x t QiXi + Q 2 Ui\l n ] 

" i=n 

min E[x^Qix n + v n u T n Q 2 u n + V n+1 \l n 



where V n is the Bellman equation at time n. This is given by 



Vn = min E 



x T n Q\x n + v n u T n Q 2 u n + V n+ x\X n 



To solve the above nested minimization problem, we assume that the solution to the functional is of the form 
V n = x'^S n (i'g~ 1 )x n + s„(j/™ -1 ), where S n and s n are functions of the past packet deliveries i>o _1 that return a 
positive semidefinite matrix and a scalar, respectively. However, both S n and s n are not functions of the applied 
control sequence Uq _i . We prove this supposition using induction. The initial condition at time N is trivially 
obtained as Vn — x n QqXn, with Sn = Qo and sjv = 0. We now assume that the functional at time n + 1 has a 
solution of the desired form, and attempt to derive this at time n. We have 



V n = min E 
= min E 



ix n + v n u^Q 2 u n + Xn +1 S n+1 (v£)x n+ i + s n+ \(u%)\l n 



Xn(Qi + A T S n+ i(u^)A)x n + v n u^{Q 2 + B T S n+1 {v$)B)u n 
+ v n x T n A T S n+1 ( v %)Bu n + iy„u^B T S n+1 (^)Ax n + w^S n+1 (us)w n + s n+1 (vs)\l n 
min x*(q i + A T E [S n+ 1 K ) K" ~ 1 ] a) x n + trace | E [S n+1 («/ ™ ) | i/? -1 ] \ + E[s n+X ) | v%~ x ] 



(V n = III/?" 



](Q 2 + B T S n+1 (v n =i,vZ- 1 )B)i 



+ x^A T S n+1 (v n =i.Mo 1 )Bu n +u^B T S n+1 (u n =i,u^ 1 )Ax r , 



(13) 



In the last equation above, the expectation of the terms preceded by v n require the conditional probability ¥{v n — 
l\fo ~ 1 ) and an evaluation of S n +i with v n = 1. The corresponding terms with v n = vanish as they are multiplied 
by v n . The control input at sample time n which minimizes the above expression is found to be u n = —L n x n , 



where the optimal control gain L n is given by (12), with k replaced by n. Substituting for u n in the functional V n , 



we get a solution to the functional of the desired form, with S n and s n given by 
SWK"- 1 ) = Qi+ A T E[S n+1 (^)\v^- 1 ]A 



(V n = IK" *) 



A T S n+ i(u n =i,uS- 1 )B(Q 3 + B T S n+1 (u n =i^ 1 )B) 1 B T S n+ i(v n =i,vS- 1 )A 



(14a) 
(14b) 



Notice that S n and s n are functions of the variables i/J 1 . When n = k, the current sample time, these variables 
are known, and S n and s n are not random. But S n +i and s„+i, for values of z G {1, . . . , ./V — n — 1}, are functions 
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of the variables i/o +,_I , of which only the variables are random variables since they are unknown to the 

controller at sample time n — k. Since the value of 5„+i is required at sample time n, we compute its conditional 
expectation as 

E [Sk+iWK -1 ] =P(i/ n = l|i/ "- 1 )5„+i( I ,„=i,<- 1 )+PK = 0|i/J- 1 )5 n+ i( I ,„=o,<- 1 ) . (14c) 

The above computation requires an evaluation of S' n _)_j(i/J ,+i_1 ) through a backward recursion for i G {1, . . . , N— n— 1} 
for all combinations of v^+i 2 . More explicitly, the expression at any time n + i, for i E {N — n — 1, . . . , 1}, is given 

by 

E [Sn+ii^- 1 )]^- 1 ] = Qi + A T E [5„ +l+1 K+')|^- 1 ]^ 

- Yl 'K+i^l,^" 1 =^Vo" _1 ) 

x A T S f n+ i + i(i/„ +i =i,^" +1_1 =£;* _1 ,i/;* _1 )i? 

x (Q 2 + -B T 5 n+i+ i( i /„ +< =i,<+ i - 1 =c*- 1 ,i/?- 1 )B) _1 

X i? T S'„+i+i(^„ + i = l,i/" +I ~ 1 =i>o~" 1 ,^o 

E [s n+l K"+ 1 - 1 )|^r 1 ] = trace {E [S'„ +i+ iK +l )|z/ r r 1 ]i? t0 } + E [v.+iK""")!^" 1 ] • 
Using the above expressions, we obtain the net cost to be 

N-l 

J = trace So-Ro + ^ trace {E[S n+ i(vg)]R w } . (15) 
n=0 

Notice that the control inputs u n are only applied to the plant and do not influence the network or v"' 1 . Thus, 
the architecture separates into a network estimator and controller, as shown in Figure [2] □ 

4.2 Comparative controllers 

In this section, we compare the performance of the FPD controller to two controllers that assume particular 
statistical models for the packet delivery sequence pdf, the IID controller and the ON controller. 
IID Controller: The IID controller was described in Schenato et al. [7] and assumes that the packet deliveries are 
i.i.d. Bernoulli with packet delivery probability equal to the a priori probability of delivering a packet through the 
network^ This is our first comparative controller, where Uk = —L l k m xk and the control gain L l k m is given by 

Lr = (Q2 + B T S^ 1 B)- 1 B T S^ 1 A . 

Here, SEi is the solution to the Riccati equation for the control problem where the packet deliveries are assumed 
to be i.i.d. Bernoulli. The backward recursion is initialized to 5™ = Qo and is given by 

SS° =Qi + A T S™A PK = l)A T S I k I + 1 B(Q 2 + BPsRBr^SgtA . 

ON Controller: The ON controller assumes that the packets are always delivered, or that the network is always 
online. This is our second comparative controller, where Uk = —L k N Xk and the control gain L k is given by 

^^(Q. + B^S^By'B^S^A . 

7 Using the stationary probability of each link under the G-E link model to calculate the end-to-end probability of delivering a packet 
through the network. 
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Here, S®^ is the solution to the Riccati equation for the classical control problem which assumes no packet losses 
on the actuation channel. The backward recursion is initialized to S N N = Qo and is given by 



c-ON 



0, + A T S^A - A T S™B(Q 2 + B T S^ 1 B)- 1 B T S^ 1 A 



Comparative Cost: The FPD controller is the most general form of a causal, optimal LQG controller that takes 
into account the packet delivery sequence pdf. It does not assume the packet delivery sequence pdf comes from 
a particular statistical model. Approximating the actual packet delivery sequence pdf with a pdf described by a 
particular statistical model, and then computing the optimal control policy, will result in a suboptimal controller. 
However, it may be less computationally expensive to obtain the control gains for such a suboptimal controller. For 
example, the IID controller and the ON controller are suboptimal controllers for networks like the one described in 
Section [2Tj since they presume a statistical model that is mismatched to the packet delivery sequence pdf obtained 
from the network model. 



Remark The average LQG cost of using a controller with control gain L c ° mv is 

N-l 

J = trace S s opt R + ^ trace {E^Jk)]^} 



(16a) 



where 



SrVo"" 1 ) = Qi + A T E [^Kd^-^ + PK = lli/f- 1 ) 



-A T S a S^<- 1 )BLl 



rcomp nT nSOpt 



-r^A 



(16b) 



and E [S^?^(u^ )\v„ i ] is computed in a similar manner to (14c). The control gain L°° mp can be the gain of a 
comparative controller (e.g., L™ or L° N ) where the statistical model for the packet delivery sequence is mismatched 
to the actual model. □ 



This can be seen from the proof of Theorem 



4.1 



if we substitute for the control input with u^ opt = — L™ mp a; n 
in (13 1, instead of minimizing the expression to find the optimal u n . On simplifying, we get the solution to the 
cost-to-go V n of the form ^S^V? -1 )^ + s^V? -1 ), with S s n opt given by (pbl and s s ° pt given by 



sopt, „- 



trace {E [S^^ur 1 ]^} +E [s^oK"] 



4.3 Algorithm to Compute Optimal Control Gain 



At sample time ft, we have u ~ . To compute given in ( 12 ), we need Sk+i(vk=i, l '$~ 1 )i which can only be obtained 
through a backward recursion from Sn- This requires knowledge of f , which are unavailable at sample time k. 
Thus, we must evaluate {Sk+x, ■ • ■ , Sn} for every possible sequence of arrivals v"' 1 ■ This algorithm is described 
below. 



1. Initialization: «S/v(i/™ 

2. for n = N- 1 : -1 : ft + 1 



JV-fe — 1 k— 1\ 



Qo, V^"-^ 1 e {0,1} 



N-k 



a) Using pc] , compute E [Sn+i^ut 1 , #T fc_1 ] , V Or"- 1 G {0, l}"- fe . 

b) Using (Il4a|, compute S , 7l (f»- 1 =f£- fc -V fc - 1 ) ) V i/?"*- 1 G {0, l}"- fc . 



3. Compute using Sk+i(vk=i,v£ 1 )- 

For ft = 0, the values So, E[Si(v )], and the other values obtained above can be used to evaluate the cost function 



according to (151 
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4.4 Computational Complexity of Optimal Control Gain 



The FPD controller is optimal but computationally expensive, as it requires an enumeration of all possible packet 
delivery sequences from the current sample time until the end of the control horizon to calculate the optimal control 



gain (12 1 at every sample time k. 



Property 2. The algorithm presented in Section \4-3\ for computing the optimal control gain for the FPD controller 
takes 0(q 3 (N — k)2 N ~ k ) operations at each sample time k, where q — max(£, m) and £ and m are the dimensions 
of the state and control vectors. 

Proof. The computational complexities below assume that multiplying aalxm matrix with amxn matrix takes 
0(£mn), and that inverting an I x I matrix takes 0(£ 3 ). 



For the computation of in (12), we need to run the algorithm presented in Section 4.3 The steps within 
the for-loop (Step |2]) of the algorithm require matrix multiplications and inversions that take 0{{2£ 3 + 6£ 2 m + 
2£ 2 + 2£m 2 + m 3 + m 2 )2 N ~ k ) operations, or 0(q 3 2 N ~ k ) operations if we let q = max(£, m). This must be repeated 
N — k — 1 times in the for-loop, so Step II takes 0(q 3 (N -k- l)2 N ~ k ). 

Finally, Step [5] takes 0{A£ 2 m + £m 2 + m 3 + m 2 ) operations for the matrix multiplications and inversions, 
which simplifies to 0(q 3 ). Combining these results and simplifying yields the computational complexity given in 
Property^ □ 

For the SIHS network model, once the network state estimates from the SIHS network estimator converge, the 
conditional probabilities / k+H-i will not change and the computations can be reused. But, for a network that 

evolves over time, like the GEIHS network model, the computations cannot be reused, and the computational cost 
remains high. 



5 Examples and Simulations 

Using the system architecture depicted in Figure [2j we will demonstrate the GEIHS network estimator on a small 
mesh network and use the packet delivery predictions in our FPD controller. Figure [6] depicts the routing topology 
and short repeating schedule of the network. Packets are generated at the source every 409 time slotsj^] and 
the packet delivery deadline is t' k — tk = 9,Vfc. The network estimator assumes all links have p u — 0.0135 and 
p d = 0.0015. 

The packet delivery predictions from the network estimator are shown in Figure [7] Although the network 
estimator provides f u k+H-i (P"^ 1 ), at each sample time k we plot the average prediction E[i>£ +H_1 ]. In this 

example, all the links are up for k <E {1, . . . , 4} and then link (3, b) fails from k — 5 onwards. After seeing a packet 
loss at k = 5, the network estimator revises its packet delivery predictions and now thinks there will be a packet 
loss at k = 7. The average prediction for the packet delivery at a particular sample time tends toward 1 or as the 
network estimator receives more information (in the form of packet deliveries) about the new state of the network. 

The prediction for k = 7 (packet generated at schedule time slot 3) at k = 5 is influenced by the packet delivery 
at k = 5 (packet generated at schedule time slot 1) because hop- by- hop routing allows the packets to traverse the 
same links under some realizations of the underlying routing topology G. Mesh networks with many interleaved 
paths allow packets generated at different schedule time slots to provide information about each others' deliveries, 
provided the links in the network have some memory. As discussed in Section |3.4[ since a packet delivery provides 
only one bit of information about the network state, it may take several packet deliveries to get good predictions 
after the network changes. 

8 Effectively, the packets are generated every 9 + AK time slots, where K is a very large integer, so we can assume slow system 
dynamics with respect to time slots and ignore the delay introduced by the network. 
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Routing Topology 
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Figure 6: Example of a simple mesh network for network estimation. 
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Figure 7: Packet delivery predictions when network in Figure [6] has all links up and then link (3,6) fails. 
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FPD, IID, and ON LQG Control Comparison 




sample time, k 




Figure 8: Plot of the different control signals and state vectors when using the FPD controller, an 11D controller, 
and an ON controller (see text for details). 



Now, consider a linear plant with the following parameters 



A = 



1.5 
1.5 



!l = W2 = 



5 
0.2 

1 
1 



, Ryj — - 



0.1 






0.1 



,Ro = 



10 






10 



10 
10 



The transfer matrix A flips and expands the components of the state at every sampling instant. The input matrix 
B requires the second component of the control input to be larger in magnitude than the first component to have 
the same effect on the respective component of the state. Also, the final state is weighted more than the other 
states in the cost criterion. We compare the three finite horizon LQG controllers discussed in Section |4j namely 



the FPD controller, the IID controller, and the ON controller with their costs ( 15 ) and ( 16a 

The controllers are connected to the plant at sample times k = 9, 10, 11 through the network example given 
in Figure [7] Figure [8] shows the control signals computed by the different controllers and the plant states when 
the control signals are applied following the actual packet delivery sequence. From the predictions at k = 8, 9, 10 
in Figure [7| we see that the FPD controller has better knowledge of the packet delivery sequence than the other 
two controllers. The FPD controller uses this knowledge to compute an optimal control signal that outputs a large 
magnitude for the second component of uio, despite the high cost of this signal. The IID and ON controllers believe 
the control packet is likely to be delivered at k — 11 and choose, instead, to output a smaller increase in the first 
component of Uu, since this will have the same effect on the final state if the control packet at k — 11 is successfully 
delivered. 

The FPD controller is better than the other controllers at driving the first component of the state close to zero 
at the end of the control horizon, k — 12. Thus, the packet delivery predictions from the network estimator help 
the FPD controller significantly lower its LQG cost, as shown in Table [l] The costs reported here are obtained 
from Monte-Carlo simulations of the system, averaged over 10,000 runs, but with the network state set to the one 
described above. 



16 



Table 1: Simulated LQG Costs (10,000 runs) for Example Described in Section[5] 



FPD Controller IID Controller 


ON Controller 


681.68 


1,008.2 


1,158.9 



6 Discussion on Network Model Selection 

The ability of the network estimator to accurately predict packet deliveries is dependent on the network model. 
A natural objection to the GEIHS network model is that it assumes links are independent and does not capture 
the full behavior of a lossy and bursty wireless link through the G-E link model [TS]. Why not use one of the 
more sophisticated link models mentioned by Willig et al. jT5]? Why not use a network model that can capture 
correlation between the links in the network? A good network model must be rich enough to capture the relevant 
behavior of the actual network, but not have too many parameters that are difficult to obtain. 

In our problem setup, the relevant behavior is the packet delivery sequence of the network. As mentioned 
in Section |3.4| the network state probability distribution docs not need to identify the exact network topology 
realization to get precise packet delivery predictions. In this regard, the GEIHS network model has too many 
states (2 E states) and may be overmodeling the actual network. However, the more relevant question is: Does the 
GEIHS network model yield accurate packet delivery predictions, predictions that are close to the actual future 
packet delivery sequence? Do the simplifications from assuming link independence and using a G-E link model 
result in inaccurate packet delivery predictions? These questions need further investigation, involving real-world 
experiments. 

Our GEIHS network model has as parameters the routing topology G, the schedule F^ T ', the G-E link transition 
probabilities {p",p^}; g £, the source a, the sink b, the packet generation times t^, and the deadlines t' k . The most 
difficult parameters to obtain are the link transition probabilities, which must be estimated by link estimators 
running on the nodes and relayed to the GEIHS network estimator. Furthermore, on a real network these parameters 
will change over time, albeit at a slower time scale than the link state changes. The issue of how to obtain these 
parameters is not addressed in this paper. 

Despite its limitations, the GEIHS network model is a good basis for comparisons when other network models 
for our problem setup are proposed in the future. It also raises several related research questions and issues. 

Are there classes of routing topologies where packet delivery statistics are less sensitive to the parameters in 
our G-E link model and pf ? How do we build networks (e.g., select routing topologies and schedules) that are 
"robust" to link modeling error and provide good packet delivery statistics (e.g., low packet loss, low delay) for 
NCSs? The latter half of the question, building networks with good packet delivery statistics, is partially addressed 
by other works in the literature like Soldati et al. |19j . which studies the problem of scheduling a network to 
optimize reliability given a routing topology and packet delivery deadline. 

Another issue arises when we use a controller that reacts to estimates of the network's state. In our problem 
setup, if the network estimator gives wrong (inaccurate) packet delivery predictions, the FPD controller can actually 
perform worse than the ON controller. How do we design FPD controllers that are robust to inaccurate packet 
delivery predictions? 

7 Conclusions 

This paper proposes two network estimators based on simple network models to characterize wireless mesh networks 
for NCSs. The goal is to obtain a better abstraction of the network, and interface to the network, to present to the 
controller and (future work) network manager. To get better performance in a NCS, the network manager needs 
to control and reconfigure the network to reduce outages and the controller needs to react to or compensate for the 
network when there are unavoidable outages. We studied a specific NCS architecture where the actuation channel 
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was over a lossy wireless mesh network and a network estimator provided packet delivery predictions for a finite 
horizon, Future-Packet-Delivery-optimized LQG controller. 

There are several directions for extending the basic problem setup in this paper, including those mentioned in 
Sections 3.3 3.4 and [6] For instance, placing the network estimator(s) on the actuators in the general system 



architecture depicted in Figure [T] is a more realistic setup but will introduce a lossy channel between the network 
estimator(s) and the controller (s). Also, one can study the use of packet delivery predictions in a receding horizon 
controller rather than a finite horizon controller. 
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